test(integration): add opt-in parallel test execution - #252
Draft
cleithner-comcast wants to merge 10 commits into
Draft
test(integration): add opt-in parallel test execution#252cleithner-comcast wants to merge 10 commits into
cleithner-comcast wants to merge 10 commits into
Conversation
cleithner-comcast
left a comment
Contributor
Author
There was a problem hiding this comment.
In addition to the below, need to look at how the suite dumps logs when an error occurs. We need sane separators if logs are interleaved so that they can be parsed.
Run the integration suite in parallel via 'py_test.sh --parallel[=N]' (serial by default so individual/interactive runs keep readable, interleaved logs). CI opts in via run_integration_tests.sh. Default workers is min(nproc/4, 64) -- roughly half the physical cores, since each worker drives a multi-threaded Barton plus a matter.js node process and nproc counts hyperthreads.
Supporting changes for concurrent test processes:
- matter: bind ephemeral operational/UDC ports under BARTON_CONFIG_MATTER_USE_RANDOM_PORT so concurrent Barton servers don't collide on CHIP_PORT/CHIP_UDC_PORT (5540/5550).
- matter: barton patch 0003 raises the CHIP commissioner's discovered-node cache 10 -> 128. On the shared mDNS plane every commissioner sees every device's advertisement, so concurrent commissionings overflowed the 10-slot cache ('Insufficient space') and discovery timed out -- capping reliable parallelism at ~4. With 128 no overflow occurs even at 64 workers.
- test env: per-process storage on tmpfs with cleanup on process exit regardless of outcome; sweep the shared compile-time ~/.brtn-ds once at session end.
- pin the fixed-port zhal mock tests (18443/8711) to one xdist worker via xdist_group + loadgroup; strip xdist's @group nodeid suffix in the subprocess-per-test runner.
- raise commission/ready/resource timeouts (5->30, 10->30) to tolerate load.
- py_test.sh: resolve repo root from script location so PYTHONPATH and LD_LIBRARY_PATH point at this worktree, not a stale primary clone.
The integration suite can now run in parallel via 'py_test.sh --parallel' (pytest-xdist). Install pytest-xdist in the builder image and bump the builder version 2.13 -> 2.14 so a fresh container and CI have the dependency.
…r sweep Per review feedback: - Restore the commission/ready/resource timeouts to their original values (5/5/10); the increases masked failures and should not have changed. - Remove the pytest_sessionfinish sweep of ~/.brtn-ds. That compile-time CHIP config dir may be shared with reference-app runs, so the test suite must not delete it; it is simply reused instead.
2.15 is reserved for the webrtc work; this parallel-test image needs its own version so the pytest-xdist install is picked up.
Under --parallel several failing tests' captured logs are reported together (and background C-library writes to fd 1/2 can interleave). Wrap each subprocess's dumped output in greppable 'BARTON TEST OUTPUT BEGIN/END [<nodeid>]' banners so the failure dump stays attributable and parseable.
cleithner-comcast
force-pushed
the
cleith/dev/testSpeedup
branch
from
August 3, 2026 16:04
6f791f2 to
330dabd
Compare
Add --client-ready-timeout / --device-added-timeout / --resource-value-timeout pytest options (defaults 5/5/30), resolved into testing/utils/timeouts.py in pytest_configure and forwarded into each per-test subprocess so the child running the test sees the same values. The wait helpers (wait_for_client_to_be_ready, wait_for_device_added, wait_for_resource_value) read them at call time. py_test.sh --parallel raises them to 30/30/30 so concurrent commissioning has headroom, while serial/interactive runs keep the fast-failing defaults for quick feedback.
Under parallel test execution, commissioning intermittently failed because the harness commissioned via the manual pairing code, which carries only a 4-bit short discriminator. CHIP's SetUpCodePairer then discovers with a kShortDiscriminator filter, matching any device sharing that short discriminator. With many devices advertising concurrently (only 16 short buckets), the commissioner frequently matched the wrong device and PASE'd to its port, which never completes -> PASE/discovery timeout. Add generate_qr_code() (full 12-bit discriminator, Base38-encoded MT: payload) alongside the existing manual pairing code logic, and commission via the QR code so the commissioner matches the exact device. Barton's ParseSetupPayload auto-detects the MT: prefix and uses the long-discriminator parser. At 64 workers the full suite goes from 15 failed / 47 passed to 62 passed.
The former nproc/4 divisor was extra headroom against commissioning timeouts that were actually the short-discriminator collision (now fixed by commissioning via the full-discriminator QR code), not CPU starvation. With that gone, one worker per physical core (nproc/2, since nproc counts hyperthreads) is reliable. Pin CI to --parallel=3: the runners have 4 cores, and a 4-core sweep showed 3 workers is ~30% faster than 2 (70s vs 100s) with zero flakes across repeated runs, while keeping a core free for the OS/dbus/otbr. The nproc/2 default would otherwise resolve to 2 on those runners.
# Conflicts: # docker/version # testing/environment/base_environment_orchestrator.py
Squashed follow-ups to the origin/main merge: - Pin GI_TYPELIB_PATH to the worktree's build/core so parallel/worktree runs load this tree's freshly-built BCore typelib. main bumped BCore to major version 4, and the container's GI_TYPELIB_PATH points at the provisioning clone, so without this a worktree resolves the wrong typelib (or fails with 'Namespace BCore not available'). - Make the side-band channel address- and lifecycle-correct: connect the client to 127.0.0.1 to match the server's bind (instead of dual-stack 'localhost'), and have the server close each connection after responding, so no kept-alive sockets linger on the loopback control channel under load. - Default to min(nproc/4, 32) workers, about one worker per two physical cores. Commissioning is a crypto-heavy PASE/CASE burst that runs concurrently on both the commissioner (Barton) and the target (a matter.js node process), so it needs ~2 physical cores while it runs; packing one worker per physical core starves the target and it misses the PASE handshake. Starvation is fixed with resources, not retries or inflated timeouts. Validated 0 flakes over 50 runs. - Specify the CI worker count in the workflow (--parallel=3) rather than in run_integration_tests.sh, since the runner's core count (GitHub-hosted runners have 4) is a property of the CI environment. The runner script now forwards its arguments to py_test.sh.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run the integration suite in parallel via 'py_test.sh --parallel[=N]' (serial by default so individual/interactive runs keep readable, interleaved logs). CI opts in via run_integration_tests.sh. Default workers is min(nproc/4, 64) -- roughly half the physical cores, since each worker drives a multi-threaded Barton plus a matter.js node process and nproc counts hyperthreads.
Supporting changes for concurrent test processes: